{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Part 1 - Introduction to Data Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#Libraries-in-Scientific-Python-Ecosystem\" data-toc-modified-id=\"Libraries-in-Scientific-Python-Ecosystem-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Libraries in Scientific Python Ecosystem</a></span><ul class=\"toc-item\"><li><span><a href=\"#Data-Processing\" data-toc-modified-id=\"Data-Processing-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Data Processing</a></span></li><li><span><a href=\"#Data-Visualization\" data-toc-modified-id=\"Data-Visualization-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Data Visualization</a></span></li><li><span><a href=\"#Data-Modeling\" data-toc-modified-id=\"Data-Modeling-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Data Modeling</a></span></li></ul></li><li><span><a href=\"#Spatially-Enabled-DataFrame\" data-toc-modified-id=\"Spatially-Enabled-DataFrame-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Spatially Enabled DataFrame</a></span><ul class=\"toc-item\"><li><span><a href=\"#What-is-a-DataFrame?\" data-toc-modified-id=\"What-is-a-DataFrame?-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>What is a DataFrame?</a></span></li><li><span><a href=\"#What-is-a-Spatially-Enabled-DataFrame-(SEDF)?\" data-toc-modified-id=\"What-is-a-Spatially-Enabled-DataFrame-(SEDF)?-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>What is a Spatially Enabled DataFrame (SEDF)?</a></span><ul class=\"toc-item\"><li><span><a href=\"#Reading-Data-into-Spatially-Enabled-DataFrame\" data-toc-modified-id=\"Reading-Data-into-Spatially-Enabled-DataFrame-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Reading Data into Spatially Enabled DataFrame</a></span></li><li><span><a href=\"#Exporting-Data-from-Spatially-Enabled-DataFrame\" data-toc-modified-id=\"Exporting-Data-from-Spatially-Enabled-DataFrame-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Exporting Data from Spatially Enabled DataFrame</a></span></li></ul></li></ul></li><li><span><a href=\"#Quick-Example\" data-toc-modified-id=\"Quick-Example-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Quick Example</a></span><ul class=\"toc-item\"><li><span><a href=\"#Read-the-Data\" data-toc-modified-id=\"Read-the-Data-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Read the Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Read-into-Spatially-Enabled-Dataframe\" data-toc-modified-id=\"Read-into-Spatially-Enabled-Dataframe-4.1.1\"><span class=\"toc-item-num\">4.1.1&nbsp;&nbsp;</span>Read into Spatially Enabled Dataframe</a></span></li><li><span><a href=\"#Plot-on-a-Map\" data-toc-modified-id=\"Plot-on-a-Map-4.1.2\"><span class=\"toc-item-num\">4.1.2&nbsp;&nbsp;</span>Plot on a Map</a></span></li></ul></li><li><span><a href=\"#Split-the-Data\" data-toc-modified-id=\"Split-the-Data-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Split the Data</a></span></li><li><span><a href=\"#Build-the-Model\" data-toc-modified-id=\"Build-the-Model-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span>Build the Model</a></span></li><li><span><a href=\"#Get-Predictions\" data-toc-modified-id=\"Get-Predictions-4.4\"><span class=\"toc-item-num\">4.4&nbsp;&nbsp;</span>Get Predictions</a></span><ul class=\"toc-item\"><li><span><a href=\"#Add-Predictions-to-Test-Data\" data-toc-modified-id=\"Add-Predictions-to-Test-Data-4.4.1\"><span class=\"toc-item-num\">4.4.1&nbsp;&nbsp;</span>Add Predictions to Test Data</a></span></li><li><span><a href=\"#Plot-on-a-Map\" data-toc-modified-id=\"Plot-on-a-Map-4.4.2\"><span class=\"toc-item-num\">4.4.2&nbsp;&nbsp;</span>Plot on a Map</a></span></li></ul></li><li><span><a href=\"#Export-Data\" data-toc-modified-id=\"Export-Data-4.5\"><span class=\"toc-item-num\">4.5&nbsp;&nbsp;</span>Export Data</a></span></li></ul></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Conclusion</a></span></li><li><span><a href=\"#References\" data-toc-modified-id=\"References-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>References</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We live in the digital era of smart devices, Internet of things (IoT), and Mobile solutions, where data has become an essential aspect of any enterprise. It is now crucial to gather, process, and analyze large volumes of data as quickly and accurately as possible.\n",
    "\n",
    "\n",
    "Python has become one of the most popular programming languages for data science, machine learning, and general software development in academia and industry. It boasts a relatively low learning curve, due to its simplicity, and a large ecosystem of data-oriented libraries that can speed up and simplify numerous tasks.\n",
    "\n",
    "\n",
    "When you are getting started, the vastness of Python may seem overwhelming, but it is not as complex as it seems. Python has also developed a large and active data analysis and scientific computing community, making it one of the most popular choices for data science. Using Python within ArcGIS enables you to easily work with open-source python libraries as well as with ArcGIS Python libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=''>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image above shows some of the popular libraries in the Python ecosystem. This is by no means a full list, as the Python ecosystem is continuously evolving with numerous other libraries. Let's look at some of the popular libraries in the scientific Python ecosystem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Libraries in Scientific Python Ecosystem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data engineering is one of the most critical and foundational skills in any data scientist’s toolkit. A data scientist needs to get the data, clean and process it, visualize the results, and then model the data to analyze and interpret trends or patterns for making critical business decisions. The availability of various multi-purpose, ready-to-use libraries to perform these tasks makes Python a top choice for analysts, researchers, and scientists alike."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Processing\n",
    "\n",
    "Data Processing is a process of cleaning and transforming data. It enables users to explore and discover useful information for decision-making. Some of the key Python libraries used for Data Processing are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1. __[NumPy](https://numpy.org/)__, short for Numerical Python, has been designed specifically for mathematical operations. It is a perfect tool for scientific computing and performing basic and advanced array operations. It primarily supports multi-dimensional arrays and vectors for complex arithmetic operations. In addition to the data structures, the library has a rich set of functions to perform algebraic operations on the supported data types. NumPy arrays form the core of nearly the entire ecosystem of data science tools in Python and are more efficient for storing and manipulating data than the other built-in Python data structures. NumPy’s high level syntax makes it accessible and productive for programmers from any background or experience level.\n",
    "\n",
    "2. __[SciPy](https://scipy.org/)__ the Scientific Python library is a collection of numerical algorithms and domain-specific toolboxes. SciPy extends capabilities of NumPy by offering advanced modules for linear algebra, integration, optimization, and statistics. Together NumPy and SciPy form a reasonably complete and mature computational foundation for many scientific computing applications.\n",
    "\n",
    "3. __[Pandas](https://pandas.pydata.org/)__ provides high-level data structures and functions designed to make working with structured or tabular data fast, easy, and expressive. Pandas blends the high-performance, array-computing ideas of NumPy with the flexible data manipulation capabilities of relational databases (such as SQL). It is based on two main data structures: \"Series\" (one-dimensional such as a list of items) and \"DataFrames\" (two-dimensional, such as a table). Pandas provides sophisticated indexing functionality to reshape, slice and dice, perform aggregations, and select subsets of data. It provides capabilities for easily handling missing data, adding/deleting columns, imputing missing data, and creating plots on the go. Pandas is a must-have tool for data wrangling and manipulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Visualization\n",
    "\n",
    "An essential function of data analysis and exploration is to visualize data. Visualization makes it easier for the human brain to detect patterns, trends, and outliers in the data. Some of the key Python libraries used for Data Visualization are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. __[Matplotlib](https://matplotlib.org/)__ is the most popular Python library for producing plots and other two-dimensional data visualizations. It can be used to generate various types of graphs such as histograms, pie-charts, or a simple bar chart. Matplotlib is highly flexible, offering the ability to customize almost every available feature. It provides an object oriented, MATLAB-like interface for users, and as such, has generally good integration with the rest of the ecosystem.\n",
    "\n",
    "2. __[Seaborn](http://seaborn.pydata.org/)__ is based on Matplotlib and serves as a useful tool for making attractive and informative statistical graphics in Python. It can be used for visualizing statistical models, summarizing data, and depicting the overall distributions. Seaborn works with the dataset as a whole and is much more intuitive than Matplotlib. It automates the creation of various plot types and creates beautiful graphics. Simply importing seaborn improves the visual aesthetics of Matplotlib plots.\n",
    "\n",
    "3. __[Bokeh](https://bokeh.org/)__ is a great tool for creating interactive and scalable visualizations inside browsers using JavaScript widgets. It is an advanced library that is fully independent of Matplotlib. Bokeh's emphasis on widgets allows users to represent the data in various formats such as graphs, plots, and labels. It empowers the user to generate elegant and concise graphics in the style of D3.js. Bokeh has the capability of high-performance interactivity over very large datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Modeling\n",
    "The process of modeling involves training a machine learning algorithm. The output from modeling is a trained model that can be used for inference and for making predictions. Some of the key Python libraries used for Data Modeling are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. __[Scikit-Learn](https://scikit-learn.org/stable/)__ has become the industry standard and is a premier general-purpose machine learning toolkit for Python programmers. Built on NumPy, SciPy and matplotlib, it features algorithms for various machine learning, statistical modeling and data mining tasks. Scikit-Learn contains submodules for tasks such as preprocessing, classification, regression, model selection, dimensionality reduction as well as clustering. The library comes with sample datasets for users to experiment.\n",
    "\n",
    "2. __[StatsModels](https://www.statsmodels.org/stable/)__ is a statistical analysis package that contains algorithms for classical statistics and econometrics. It provides a complement to scipy for statistical computations and is more focused on providing statistical inference, uncertainty estimates and p-values for parameters. StatsModels features submodules for various tasks such as Regression Analysis, Analysis of Variance (ANOVA), Time Series Analysis, Non-parametric methods and Visualization of model results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this guide series, we will focus on two key libraries in the scientific Python ecosystem that are used for data processing, __NumPy__ and __Pandas__. Before we go into the details of these two topics, we will briefly discuss `Spatially Enabled DataFrame`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spatially Enabled DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a DataFrame?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html) represents a rectangular table of data and contains an ordered collection of columns. You can think of it as a spreadsheet or SQL table where each column has a column name for reference, and each row can be accessed by using row numbers. Column names and row numbers are known as column and row indexes.\n",
    "\n",
    "DataFrame is a fundamental __Pandas__ data structure in which each column can be of a different value type (numeric, string, boolean, etc.). A data set can be first read into a DataFrame, and then various operations (i.e. indexing, grouping, aggregation etc.) can be easily applied to it.\n",
    "\n",
    "Given some data, let's look at how a dataset can be read into a DataFrame to see what a DataFrame looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data Creation\n",
    "data = {'state':['CA','WA','CA','WA','CA','WA'],\n",
    "        'year':[2015,2015,2016,2016,2017,2017],\n",
    "        'population':[3.5,2.5,4.5,3.0,5.0,3.25]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>year</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CA</td>\n",
       "      <td>2015</td>\n",
       "      <td>3.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>WA</td>\n",
       "      <td>2015</td>\n",
       "      <td>2.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CA</td>\n",
       "      <td>2016</td>\n",
       "      <td>4.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>WA</td>\n",
       "      <td>2016</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CA</td>\n",
       "      <td>2017</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>WA</td>\n",
       "      <td>2017</td>\n",
       "      <td>3.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  state  year  population\n",
       "0    CA  2015        3.50\n",
       "1    WA  2015        2.50\n",
       "2    CA  2016        4.50\n",
       "3    WA  2016        3.00\n",
       "4    CA  2017        5.00\n",
       "5    WA  2017        3.25"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read data into a dataframe\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You can see the tabular structure of data with indexed rows and columns. We will dive deeper into DataFrame in the `Introduction to Pandas` part of the guide series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a Spatially Enabled DataFrame (SEDF)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [`Spatially Enabled DataFrame`](/python/api-reference/arcgis.features.toc.html#arcgis.features.GeoAccessor) (SEDF) inserts _\"spatial abilities\"_ into the popular [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html#dataframe). This allows users to use intuitive Pandas operations on both the attribute and spatial columns. With SEDF, you can easily manipulate geometric and attribute data. SEDF is a capability that is added to the Pandas DataFrame structure, by the ArcGIS API for Python, to give it spatial abilities. \n",
    "\n",
    "SEDF is based on data structures inherently suited to data analysis, with natural operations for the filtering and inspecting of subsets of values, which are fundamental to statistical and geographic manipulations.\n",
    "\n",
    "Let's quickly look at how data can be imported and exported using __Spatially Enabled DataFrame__. The details shown below are a high level overview and we will take a deeper dive into working with Spatially Enabled DataFrame in the later parts of this guide series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reading Data into Spatially Enabled DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Spatially Enabled DataFrame` (SEDF) can read data from many sources, including:\n",
    "- [Shapefiles](https://doc.arcgis.com/en/arcgis-online/reference/shapefiles.htm)\n",
    "- [Pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html)\n",
    "- [Feature classes](https://pro.arcgis.com/en/pro-app/latest/help/data/feature-classes/feature-classes.htm)\n",
    "- [GeoJSON](https://geojson.org/)\n",
    "- [Feature Layer](https://doc.arcgis.com/en/arcgis-online/manage-data/hosted-web-layers.htm)\n",
    "\n",
    "SEDF integrates with Esri's [`ArcPy` site-package](https://pro.arcgis.com/en/pro-app/arcpy/get-started/what-is-arcpy-.htm) as well as with the open source [pyshp](https://github.com/GeospatialPython/pyshp/), [shapely](https://github.com/Toblerity/Shapely) and [fiona](https://github.com/Toblerity/Fiona) packages. This means that SEDF can use either of these geometry engines to provide you options for easily working with geospatial data regardless of your platform. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exporting Data from Spatially Enabled DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SEDF can export data to various data formats for use in other applications. Export options are:\n",
    "\n",
    "- [Feature Layers](https://doc.arcgis.com/en/arcgis-online/share-maps/hosted-web-layers.htm)\n",
    "- [Feature Collections](https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html#featurelayercollection)\n",
    "- [Feature Set](https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html#featureset)\n",
    "- [GeoJSON](http://geojson.org/)\n",
    "- [Feature Class](https://pro.arcgis.com/en/pro-app/latest/help/data/feature-classes/feature-classes.htm)\n",
    "- [Pickle](https://pythontips.com/2013/08/02/what-is-pickle-in-python/)\n",
    "- [HDF](https://support.hdfgroup.org/HDF5/Tutor/HDF5Intro.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at an example of utilizing `Spatially Enabled DataFrame` (SEDF) through the machine learning lifecycle. We will __focus on the usage of SEDF__ through the process and not so much on the intepretation or results of the model. The example shows how to:\n",
    "- Read data from Pandas DataFrame into a SEDF.\n",
    "- Use SEDF with other libraries in python ecosystem for modeling and predictions.\n",
    "- Merge modeled results back into SEDF.\n",
    "- Export data from a SEDF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a subset of [Covid-19 Nursing Home data](https://data.cms.gov/Special-Programs-Initiatives-COVID-19-Nursing-Home/COVID-19-Nursing-Home-Dataset/s2uc-8wxp) from Centers for Medicare & Medicaid Services ([CMS](https://www.cms.gov/)) to illustrate this example. __Note__ that the dataset used in this example has been curated for illustration purposes and does not reflect the complete dataset available at [CMS](https://www.cms.gov/) website.\n",
    "\n",
    "__Goal:__ Predict \"Total Number of Occupied Beds\" using other variables in the data. \n",
    "\n",
    "\n",
    "\n",
    "In this example, we will:\n",
    "1. Read the data (with location attributes) into a SEDF and plot SEDF on a map.\n",
    "2. Split SEDF into train and test sets.\n",
    "3. Build a linear regression model on training data.\n",
    "4. Get Predictions for the model using test data.\n",
    "5. Add Predictions back to SEDF.\n",
    "6. Plot SEDF with predicted data on a map.\n",
    "7. Export SEDF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Libraries\n",
    "from IPython.display import display\n",
    "\n",
    "import pandas as pd\n",
    "from arcgis.gis import GIS\n",
    "import geopandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a GIS Connection\n",
    "gis = GIS(profile='your_online_profile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Provider Name</th>\n",
       "      <th>Provider City</th>\n",
       "      <th>Provider State</th>\n",
       "      <th>Residents Weekly Admissions COVID-19</th>\n",
       "      <th>Residents Total Admissions COVID-19</th>\n",
       "      <th>Residents Weekly Confirmed COVID-19</th>\n",
       "      <th>Residents Total Confirmed COVID-19</th>\n",
       "      <th>Residents Weekly Suspected COVID-19</th>\n",
       "      <th>Residents Total Suspected COVID-19</th>\n",
       "      <th>Residents Weekly All Deaths</th>\n",
       "      <th>Residents Total All Deaths</th>\n",
       "      <th>Residents Weekly COVID-19 Deaths</th>\n",
       "      <th>Residents Total COVID-19 Deaths</th>\n",
       "      <th>Number of All Beds</th>\n",
       "      <th>Total Number of Occupied Beds</th>\n",
       "      <th>LONGITUDE</th>\n",
       "      <th>LATITUDE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GROSSE POINTE MANOR</td>\n",
       "      <td>NILES</td>\n",
       "      <td>IL</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>99</td>\n",
       "      <td>61</td>\n",
       "      <td>-87.792973</td>\n",
       "      <td>42.012012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MILLER'S MERRY MANOR</td>\n",
       "      <td>DUNKIRK</td>\n",
       "      <td>IN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>46</td>\n",
       "      <td>43</td>\n",
       "      <td>-85.197651</td>\n",
       "      <td>40.392722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>PARKWAY MANOR</td>\n",
       "      <td>MARION</td>\n",
       "      <td>IL</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>131</td>\n",
       "      <td>84</td>\n",
       "      <td>-88.982944</td>\n",
       "      <td>37.750143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AVANTARA LONG GROVE</td>\n",
       "      <td>LONG GROVE</td>\n",
       "      <td>IL</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>141</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>195</td>\n",
       "      <td>131</td>\n",
       "      <td>-87.986442</td>\n",
       "      <td>42.160843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HARMONY NURSING &amp; REHAB CENTER</td>\n",
       "      <td>CHICAGO</td>\n",
       "      <td>IL</td>\n",
       "      <td>2</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>180</td>\n",
       "      <td>116</td>\n",
       "      <td>-87.726353</td>\n",
       "      <td>41.975505</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Provider Name Provider City Provider State  \\\n",
       "0             GROSSE POINTE MANOR         NILES             IL   \n",
       "1            MILLER'S MERRY MANOR       DUNKIRK             IN   \n",
       "2                   PARKWAY MANOR        MARION             IL   \n",
       "3             AVANTARA LONG GROVE    LONG GROVE             IL   \n",
       "4  HARMONY NURSING & REHAB CENTER       CHICAGO             IL   \n",
       "\n",
       "   Residents Weekly Admissions COVID-19  Residents Total Admissions COVID-19  \\\n",
       "0                                     3                                    5   \n",
       "1                                     0                                    0   \n",
       "2                                     0                                    0   \n",
       "3                                     1                                    6   \n",
       "4                                     2                                   19   \n",
       "\n",
       "   Residents Weekly Confirmed COVID-19  Residents Total Confirmed COVID-19  \\\n",
       "0                                   10                                  56   \n",
       "1                                    0                                   0   \n",
       "2                                    0                                   0   \n",
       "3                                    0                                 141   \n",
       "4                                    1                                  75   \n",
       "\n",
       "   Residents Weekly Suspected COVID-19  Residents Total Suspected COVID-19  \\\n",
       "0                                    0                                  10   \n",
       "1                                    0                                   0   \n",
       "2                                    0                                   0   \n",
       "3                                    3                                   3   \n",
       "4                                    0                                   0   \n",
       "\n",
       "   Residents Weekly All Deaths  Residents Total All Deaths  \\\n",
       "0                            6                          15   \n",
       "1                            0                           0   \n",
       "2                            0                           0   \n",
       "3                            0                           0   \n",
       "4                            0                          43   \n",
       "\n",
       "   Residents Weekly COVID-19 Deaths  Residents Total COVID-19 Deaths  \\\n",
       "0                                 4                               12   \n",
       "1                                 0                                0   \n",
       "2                                 0                                0   \n",
       "3                                 0                                0   \n",
       "4                                 0                               16   \n",
       "\n",
       "   Number of All Beds  Total Number of Occupied Beds  LONGITUDE   LATITUDE  \n",
       "0                  99                             61 -87.792973  42.012012  \n",
       "1                  46                             43 -85.197651  40.392722  \n",
       "2                 131                             84 -88.982944  37.750143  \n",
       "3                 195                            131 -87.986442  42.160843  \n",
       "4                 180                            116 -87.726353  41.975505  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the data\n",
    "df = pd.read_csv('../data/sample_cms_data.csv')\n",
    "\n",
    "# Return the first 5 records\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 124 entries, 0 to 123\n",
      "Data columns (total 17 columns):\n",
      " #   Column                                Non-Null Count  Dtype  \n",
      "---  ------                                --------------  -----  \n",
      " 0   Provider Name                         124 non-null    object \n",
      " 1   Provider City                         124 non-null    object \n",
      " 2   Provider State                        124 non-null    object \n",
      " 3   Residents Weekly Admissions COVID-19  124 non-null    int64  \n",
      " 4   Residents Total Admissions COVID-19   124 non-null    int64  \n",
      " 5   Residents Weekly Confirmed COVID-19   124 non-null    int64  \n",
      " 6   Residents Total Confirmed COVID-19    124 non-null    int64  \n",
      " 7   Residents Weekly Suspected COVID-19   124 non-null    int64  \n",
      " 8   Residents Total Suspected COVID-19    124 non-null    int64  \n",
      " 9   Residents Weekly All Deaths           124 non-null    int64  \n",
      " 10  Residents Total All Deaths            124 non-null    int64  \n",
      " 11  Residents Weekly COVID-19 Deaths      124 non-null    int64  \n",
      " 12  Residents Total COVID-19 Deaths       124 non-null    int64  \n",
      " 13  Number of All Beds                    124 non-null    int64  \n",
      " 14  Total Number of Occupied Beds         124 non-null    int64  \n",
      " 15  LONGITUDE                             124 non-null    float64\n",
      " 16  LATITUDE                              124 non-null    float64\n",
      "dtypes: float64(2), int64(12), object(3)\n",
      "memory usage: 16.6+ KB\n"
     ]
    }
   ],
   "source": [
    "# Get concise summary of the dataframe\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains 124 records and 17 columns. Each record represents a nursing home in the states of Indiana and Illinois. Each column contains information about the nursing home such as:\n",
    "- Name of the nursing home, its city and its state\n",
    "- Details of resident Covid cases, deaths and number of beds\n",
    "- Location of nursing home as Latitude and Longitude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read into Spatially Enabled Dataframe\n",
    "\n",
    "Any Pandas DataFrame with location information (Latitude and Longitude) can be read into a Spatially Enabled DataFrame using the `from_xy()` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Provider Name</th>\n",
       "      <th>Provider City</th>\n",
       "      <th>Provider State</th>\n",
       "      <th>Residents Weekly Admissions COVID-19</th>\n",
       "      <th>Residents Total Admissions COVID-19</th>\n",
       "      <th>Residents Weekly Confirmed COVID-19</th>\n",
       "      <th>Residents Total Confirmed COVID-19</th>\n",
       "      <th>Residents Weekly Suspected COVID-19</th>\n",
       "      <th>Residents Total Suspected COVID-19</th>\n",
       "      <th>Residents Weekly All Deaths</th>\n",
       "      <th>Residents Total All Deaths</th>\n",
       "      <th>Residents Weekly COVID-19 Deaths</th>\n",
       "      <th>Residents Total COVID-19 Deaths</th>\n",
       "      <th>Number of All Beds</th>\n",
       "      <th>Total Number of Occupied Beds</th>\n",
       "      <th>LONGITUDE</th>\n",
       "      <th>LATITUDE</th>\n",
       "      <th>SHAPE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GROSSE POINTE MANOR</td>\n",
       "      <td>NILES</td>\n",
       "      <td>IL</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>99</td>\n",
       "      <td>61</td>\n",
       "      <td>-87.792973</td>\n",
       "      <td>42.012012</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -87....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MILLER'S MERRY MANOR</td>\n",
       "      <td>DUNKIRK</td>\n",
       "      <td>IN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>46</td>\n",
       "      <td>43</td>\n",
       "      <td>-85.197651</td>\n",
       "      <td>40.392722</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>PARKWAY MANOR</td>\n",
       "      <td>MARION</td>\n",
       "      <td>IL</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>131</td>\n",
       "      <td>84</td>\n",
       "      <td>-88.982944</td>\n",
       "      <td>37.750143</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -88....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AVANTARA LONG GROVE</td>\n",
       "      <td>LONG GROVE</td>\n",
       "      <td>IL</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>141</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>195</td>\n",
       "      <td>131</td>\n",
       "      <td>-87.986442</td>\n",
       "      <td>42.160843</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -87....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HARMONY NURSING &amp; REHAB CENTER</td>\n",
       "      <td>CHICAGO</td>\n",
       "      <td>IL</td>\n",
       "      <td>2</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>180</td>\n",
       "      <td>116</td>\n",
       "      <td>-87.726353</td>\n",
       "      <td>41.975505</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -87....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Provider Name Provider City Provider State  \\\n",
       "0             GROSSE POINTE MANOR         NILES             IL   \n",
       "1            MILLER'S MERRY MANOR       DUNKIRK             IN   \n",
       "2                   PARKWAY MANOR        MARION             IL   \n",
       "3             AVANTARA LONG GROVE    LONG GROVE             IL   \n",
       "4  HARMONY NURSING & REHAB CENTER       CHICAGO             IL   \n",
       "\n",
       "   Residents Weekly Admissions COVID-19  Residents Total Admissions COVID-19  \\\n",
       "0                                     3                                    5   \n",
       "1                                     0                                    0   \n",
       "2                                     0                                    0   \n",
       "3                                     1                                    6   \n",
       "4                                     2                                   19   \n",
       "\n",
       "   Residents Weekly Confirmed COVID-19  Residents Total Confirmed COVID-19  \\\n",
       "0                                   10                                  56   \n",
       "1                                    0                                   0   \n",
       "2                                    0                                   0   \n",
       "3                                    0                                 141   \n",
       "4                                    1                                  75   \n",
       "\n",
       "   Residents Weekly Suspected COVID-19  Residents Total Suspected COVID-19  \\\n",
       "0                                    0                                  10   \n",
       "1                                    0                                   0   \n",
       "2                                    0                                   0   \n",
       "3                                    3                                   3   \n",
       "4                                    0                                   0   \n",
       "\n",
       "   Residents Weekly All Deaths  Residents Total All Deaths  \\\n",
       "0                            6                          15   \n",
       "1                            0                           0   \n",
       "2                            0                           0   \n",
       "3                            0                           0   \n",
       "4                            0                          43   \n",
       "\n",
       "   Residents Weekly COVID-19 Deaths  Residents Total COVID-19 Deaths  \\\n",
       "0                                 4                               12   \n",
       "1                                 0                                0   \n",
       "2                                 0                                0   \n",
       "3                                 0                                0   \n",
       "4                                 0                               16   \n",
       "\n",
       "   Number of All Beds  Total Number of Occupied Beds  LONGITUDE   LATITUDE  \\\n",
       "0                  99                             61 -87.792973  42.012012   \n",
       "1                  46                             43 -85.197651  40.392722   \n",
       "2                 131                             84 -88.982944  37.750143   \n",
       "3                 195                            131 -87.986442  42.160843   \n",
       "4                 180                            116 -87.726353  41.975505   \n",
       "\n",
       "                                               SHAPE  \n",
       "0  {\"spatialReference\": {\"wkid\": 4326}, \"x\": -87....  \n",
       "1  {\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....  \n",
       "2  {\"spatialReference\": {\"wkid\": 4326}, \"x\": -88....  \n",
       "3  {\"spatialReference\": {\"wkid\": 4326}, \"x\": -87....  \n",
       "4  {\"spatialReference\": {\"wkid\": 4326}, \"x\": -87....  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sedf = pd.DataFrame.spatial.from_xy(df,'LONGITUDE','LATITUDE')\n",
    "sedf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Spatially Enabled DataFrame` (SEDF) adds spatial abilities to the data. A `SHAPE` column gets added to the dataset as it is read into a  SEDF. We can now plot this DataFrame on a map."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot on a Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m1 = gis.map('IL, USA')\n",
    "m1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Points displayed on the map show the location of each nursing home in our data. Clicking on a point displays attribute information for that nursing home."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sedf.spatial.plot(m1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the Data\n",
    "\n",
    "We will split the `Spatially Enabled DataFrame` into training and test datasets and separate out the predictor and response variables in training and test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data into Train and Test Sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "train_data, test_data = train_test_split(sedf, test_size=0.2, random_state=101)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of training data: (99, 18)\n",
      "Shape of testing data: (25, 18)\n"
     ]
    }
   ],
   "source": [
    "# Look at shape of training and test datasets\n",
    "print(f'Shape of training data: {train_data.shape}')\n",
    "print(f'Shape of testing data: {test_data.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Response Variable__    \n",
    "Any regression prediction task requires a variable of interest, a variable we would like to predict. This variable is called as a `Response` variable, also referred to as __y__ variable or __Dependent__ variable. Our __goal__ is to predict \"Total Number of Occupied Beds\", so our __y__ variable will be \"Total Number of Occupied Beds\". \n",
    "\n",
    "__Predictor Variables__    \n",
    "All other variables the affect the Response variable are called `Predictor` variables. These predictor variables are also known as __x__ variables or __Independent__ variables. In this example, we will use only numerical variables related to Covid cases, deaths and number of beds as __x__ variables, and we will ignore provder details such as name, city, state or location information. \n",
    "\n",
    "Here, we use __Indexing__ to select specific columns from the DataFrame. We will talk about __Indexing__ in more detail in the later sections of this guide series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Separate predictors and response variables for train and test data\n",
    "train_x = train_data.iloc[:,3:-4]\n",
    "train_y = train_data.iloc[:,-4]\n",
    "test_x = test_data.iloc[:,3:-4]\n",
    "test_y = test_data.iloc[:,-4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Model\n",
    "We will build and fit a Linear Regression model using the `LinearRegression()` method from the Scikit-learn library. Our goal is to predict the total number of occupied beds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model\n",
    "from sklearn import linear_model\n",
    "\n",
    "# Create linear regression object\n",
    "lr_model = linear_model.LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train the model using the training sets\n",
    "lr_model.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Predictions\n",
    "We will now use the model to make predictions on our test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 70.18799777,  79.35734213,  40.52267526, 112.32693137,\n",
       "        74.56730982,  92.59096106,  70.69189401,  29.84238321,\n",
       "       108.09537913,  81.10718742,  59.90388811,  67.44325594,\n",
       "        70.62977058,  96.44880679,  85.19537597,  39.10578923,\n",
       "        63.88519971,  76.36549693,  38.94543793,  41.96507956,\n",
       "        50.41997091,  66.00665849,  33.30750881,  75.17989671,\n",
       "        63.09585712])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get predictions\n",
    "bed_predictions = lr_model.predict(test_x)\n",
    "bed_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add Predictions to Test Data\n",
    "\n",
    "Here, we add predictions back to the test data as a new column, `Predicted_Occupied_Beds`. Since the test dataset is a __Spatially Enabled DataFrame__, it continues to provide spatial abilities to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted_Occupied_Beds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>70.187998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>79.357342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>40.522675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>112.326931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>74.567310</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted_Occupied_Beds\n",
       "74                 70.187998\n",
       "123                79.357342\n",
       "78                 40.522675\n",
       "41                112.326931\n",
       "79                 74.567310"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert predictions into a dataframe\n",
    "pred_available_beds = pd.DataFrame(bed_predictions, index = test_data.index, \n",
    "                                   columns=['Predicted_Occupied_Beds'])\n",
    "pred_available_beds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Provider Name</th>\n",
       "      <th>Provider City</th>\n",
       "      <th>Provider State</th>\n",
       "      <th>Residents Weekly Admissions COVID-19</th>\n",
       "      <th>Residents Total Admissions COVID-19</th>\n",
       "      <th>Residents Weekly Confirmed COVID-19</th>\n",
       "      <th>Residents Total Confirmed COVID-19</th>\n",
       "      <th>Residents Weekly Suspected COVID-19</th>\n",
       "      <th>Residents Total Suspected COVID-19</th>\n",
       "      <th>Residents Weekly All Deaths</th>\n",
       "      <th>Residents Total All Deaths</th>\n",
       "      <th>Residents Weekly COVID-19 Deaths</th>\n",
       "      <th>Residents Total COVID-19 Deaths</th>\n",
       "      <th>Number of All Beds</th>\n",
       "      <th>Total Number of Occupied Beds</th>\n",
       "      <th>LONGITUDE</th>\n",
       "      <th>LATITUDE</th>\n",
       "      <th>SHAPE</th>\n",
       "      <th>Predicted_Occupied_Beds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>GOLDEN YEARS HOMESTEAD</td>\n",
       "      <td>FORT WAYNE</td>\n",
       "      <td>IN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>110</td>\n",
       "      <td>104</td>\n",
       "      <td>-85.036651</td>\n",
       "      <td>41.107479</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....</td>\n",
       "      <td>70.187998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>WATERS OF DILLSBORO-ROSS MANOR, THE</td>\n",
       "      <td>DILLSBORO</td>\n",
       "      <td>IN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>123</td>\n",
       "      <td>75</td>\n",
       "      <td>-85.056649</td>\n",
       "      <td>39.018794</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....</td>\n",
       "      <td>79.357342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>TOWNE HOUSE RETIREMENT COMMUNITY</td>\n",
       "      <td>FORT WAYNE</td>\n",
       "      <td>IN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>58</td>\n",
       "      <td>46</td>\n",
       "      <td>-85.111952</td>\n",
       "      <td>41.133477</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....</td>\n",
       "      <td>40.522675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>UNIVERSITY HEIGHTS HEALTH AND LIVING COMMUNITY</td>\n",
       "      <td>INDIANAPOLIS</td>\n",
       "      <td>IN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>174</td>\n",
       "      <td>133</td>\n",
       "      <td>-86.135442</td>\n",
       "      <td>39.635530</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -86....</td>\n",
       "      <td>112.326931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>SHARON HEALTH CARE PINES</td>\n",
       "      <td>PEORIA</td>\n",
       "      <td>IL</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>116</td>\n",
       "      <td>96</td>\n",
       "      <td>-89.643629</td>\n",
       "      <td>40.731764</td>\n",
       "      <td>{\"spatialReference\": {\"wkid\": 4326}, \"x\": -89....</td>\n",
       "      <td>74.567310</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      Provider Name Provider City  \\\n",
       "74                           GOLDEN YEARS HOMESTEAD    FORT WAYNE   \n",
       "123             WATERS OF DILLSBORO-ROSS MANOR, THE     DILLSBORO   \n",
       "78                 TOWNE HOUSE RETIREMENT COMMUNITY    FORT WAYNE   \n",
       "41   UNIVERSITY HEIGHTS HEALTH AND LIVING COMMUNITY  INDIANAPOLIS   \n",
       "79                         SHARON HEALTH CARE PINES        PEORIA   \n",
       "\n",
       "    Provider State  Residents Weekly Admissions COVID-19  \\\n",
       "74              IN                                     0   \n",
       "123             IN                                     0   \n",
       "78              IN                                     0   \n",
       "41              IN                                     0   \n",
       "79              IL                                     0   \n",
       "\n",
       "     Residents Total Admissions COVID-19  Residents Weekly Confirmed COVID-19  \\\n",
       "74                                     5                                    0   \n",
       "123                                    4                                    0   \n",
       "78                                     0                                    1   \n",
       "41                                     0                                    0   \n",
       "79                                     0                                    0   \n",
       "\n",
       "     Residents Total Confirmed COVID-19  Residents Weekly Suspected COVID-19  \\\n",
       "74                                    0                                    0   \n",
       "123                                   0                                    0   \n",
       "78                                    1                                    1   \n",
       "41                                    0                                    0   \n",
       "79                                    0                                    0   \n",
       "\n",
       "     Residents Total Suspected COVID-19  Residents Weekly All Deaths  \\\n",
       "74                                    0                            0   \n",
       "123                                   0                            0   \n",
       "78                                    1                            0   \n",
       "41                                    0                            0   \n",
       "79                                    0                            0   \n",
       "\n",
       "     Residents Total All Deaths  Residents Weekly COVID-19 Deaths  \\\n",
       "74                            0                                 0   \n",
       "123                           7                                 0   \n",
       "78                            1                                 0   \n",
       "41                            8                                 0   \n",
       "79                            1                                 0   \n",
       "\n",
       "     Residents Total COVID-19 Deaths  Number of All Beds  \\\n",
       "74                                 0                 110   \n",
       "123                                0                 123   \n",
       "78                                 0                  58   \n",
       "41                                 0                 174   \n",
       "79                                 0                 116   \n",
       "\n",
       "     Total Number of Occupied Beds  LONGITUDE   LATITUDE  \\\n",
       "74                             104 -85.036651  41.107479   \n",
       "123                             75 -85.056649  39.018794   \n",
       "78                              46 -85.111952  41.133477   \n",
       "41                             133 -86.135442  39.635530   \n",
       "79                              96 -89.643629  40.731764   \n",
       "\n",
       "                                                 SHAPE  \\\n",
       "74   {\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....   \n",
       "123  {\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....   \n",
       "78   {\"spatialReference\": {\"wkid\": 4326}, \"x\": -85....   \n",
       "41   {\"spatialReference\": {\"wkid\": 4326}, \"x\": -86....   \n",
       "79   {\"spatialReference\": {\"wkid\": 4326}, \"x\": -89....   \n",
       "\n",
       "     Predicted_Occupied_Beds  \n",
       "74                 70.187998  \n",
       "123                79.357342  \n",
       "78                 40.522675  \n",
       "41                112.326931  \n",
       "79                 74.567310  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add predictions back to test dataset\n",
    "test_data = pd.concat([test_data, pred_available_beds], axis=1)\n",
    "test_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot on a Map\n",
    "\n",
    "Here, we plot `test_data` on a map. The map shows the location of each nursing home in the test dataset, along with the attribute information. We can see model prediction results added as `Predicted_Occupied_Beds` column, along with the actual number of occupied beds, `Total_Number_of_Occupied_Beds` , in the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m2 = gis.map('IL, USA')\n",
    "m2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.spatial.plot(m2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Export Data\n",
    "\n",
    "We will now export the Spatially Enabled DataFrame `test_data` to a feature layer. The `to_featurelayer()` method allows us to publish spatially enabled DataFrame as feature layers to the portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://geosaurus.maps.arcgis.com/home/item.html?id=070693a6ab1f41648dc2e297e989eb70' target='_blank'>\n",
       "                        <img src='http://static.arcgis.com/images/desktopapp.png' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\"     style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://geosaurus.maps.arcgis.com/home/item.html?id=070693a6ab1f41648dc2e297e989eb70' target='_blank'><b>sedf_predictions</b>\n",
       "                        </a>\n",
       "                        <br/><img src='https://geosaurus.maps.arcgis.com/home/js/jsapi/esri/css/images/item_type_icons/featureshosted16.png' style=\"vertical-align:middle;\">Feature Layer Collection by arcgis_python\n",
       "                        <br/>Last Modified: November 25, 2020\n",
       "                        <br/>0 comments, 0 views\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Item title:\"sedf_predictions\" type:Feature Layer Collection owner:arcgis_python>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lyr = test_data.spatial.to_featurelayer('sedf_predictions')\n",
    "lyr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are numerous libraries in the scientific python ecosystem. In this part of the guide series, we briefly discussed some of the key libraries used for data processing, visualization, and modeling. We introduced the concept of the __Spatially Enabled DataFrame__ (SEDF) and how it adds \"spatial\" abilities to the data. You have also seen an end-to-end example of using SEDF through the machine learning lifecycle, starting from reading data into SEDF, to exporting a SEDF.\n",
    "\n",
    "\n",
    "In the next part of this guide series, you will learn more about [NumPy](https://numpy.org/) in the Introduction to NumPy section.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Wes McKinney. 2017. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython (2nd. ed.). O'Reilly Media, Inc.    \n",
    "    \n",
    "[2] Jake VanderPlas. 2016. Python Data Science Handbook: Essential Tools for Working with Data (1st. ed.). O'Reilly Media, Inc."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "livereveal": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "244px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
